1. Field of the Invention
The present invention relates to a stereo calibration apparatus for acquiring the image transformation parameters between a plurality of image pickup devices, and a stereo image monitoring apparatus using the same stereo calibration apparatus.
2. Description of the Related Art
In an image monitor for automatically detecting a suspicious person, employing an image pickup device such as a CCD camera, and an active safety device or a vehicle guidance system for finding an obstacle on the road or recognizing the traffic lane, employing an image pickup device mounted on the vehicle, a higher detection performance can be easily attained by monitoring a common field of view, employing a plurality of image pickup devices rather than one image pickup device.
The reason is that there are virtually an infinite number of variations for the target such as an invader or obstacle, and that it is actually impossible to obtain the specifications encompassing all the variations. Hence, in the environment where there is a false detection factor such as a texture similar to the target or an unexpected image change on the road surface, it is very difficult to correctly distinguish the target based on an image pattern alone. On the other hand, if a plurality of image pickup devices can be used, information regarding the three dimensional position or structure of the object in the common field of view is obtained on the basis of a stereoscopic principle. This makes it possible to easily distinguish the target from the false detection factor.
In order to obtain the three dimensional information of the object accurately, a stereoscopic matching problem must be solved for all positions in an image, but is very difficult to solve, especially when there are complex and many environmental disturbances such as in the outdoor environment. However, in cases of using a method as described in JP-A-Hei. 11-328365or JP-A-2000-293693, although precise three dimensional information on a structure of an object is not obtained, it becomes possible to correctly detect only an image area having a height with respect to a certain plane in the three dimensional space, which is used as a reference, such as the ground or road surface. Even in the outdoor environment, it becomes possible to detect presence/absence of a target and a position of the target precisely.
The principle of these methods is as follows.
Suppose now that two image pickup devices of cameras are installed to monitor a monitoring region as the common field of view. The monitoring region without invader or obstacle is assumed to be a plane within the three dimensional space such as the road surface, which is called a reference plane. If a point X on the reference plane is photographed by two cameras at the same time, where an image of X photographed by one camera has the image coordinates (x, y) and an image of X photographed by the other camera has the image coordinates (x′, y′), and the cameras have the focal length f and f′, respectively. A transformation from one camera image to the other is the projection transformation, represented by the following relation.
                              x          ′                =                                            f              ′                        ⁢                                                                                H                    11                                    ⁢                  x                                +                                                      H                    12                                    ⁢                  y                                +                                                      H                    13                                    ⁢                  f                                                                                                  H                    31                                    ⁢                  x                                +                                                      H                    32                                    ⁢                  y                                +                                                      H                    33                                    ⁢                  f                                                      ⁢                                                  ⁢                          y              ′                                =                                    f              ′                        ⁢                                                                                H                    21                                    ⁢                  x                                +                                                      H                    22                                    ⁢                  y                                +                                                      H                    23                                    ⁢                  f                                                                                                  H                    31                                    ⁢                  x                                +                                                      H                    32                                    ⁢                  y                                +                                                      H                    33                                    ⁢                  f                                                                                        (        1        )            Herein, supposing that the coordinates of each image are represented as X=(x, y, f)T and X′=(x′, y′, f′)T, employing a homogeneous coordinate representation (T denotes a transposition), and H is a 3×3 matrix consisting of H11, . . . , H33 elements, the image transformation X to X′ is represented such as,sx′=Hx  (2)where s is a non-zero constant. Since s depends on X and H has the degree of freedom 8, H is represented without losing generality as follows.
                    ℍ        =                  (                                                                      H                  11                                                                              H                  12                                                                              H                  13                                                                                                      H                  21                                                                              H                  22                                                                              H                  23                                                                                                      H                  31                                                                              H                  32                                                            1                                              )                                    (        3        )            As far as X is on the reference plane, H can be obtained as constant irrespective of X. H is called a transformation matrix. When the transformation matrix H is known, the transformation X to X′ is calculated in accordance with the expression (2). The image feature (brightness value) near the coordinates (x, y) of one camera image and the image feature near the coordinates (x′, y′) of the other camera image are compared. If both are identical, it is judged that X is on the reference plane. On the contrary, if both the image features are different, it is judged that X is not on the reference plane, that is, some invader or obstacle having a height with respect to the reference plane exists at the position (x, y) or (x′, y′) in the image.
In this manner, if H is calculated in advance, it is possible to determine whether or not any object having a height is on the reference plane, and a high performance image monitor without falsely detecting any texture or object image on the road surface is realized.
A conventional method for calculating H will be described below.
If four or more sets of corresponding points (pixels) between two camera images {Xm=(xm, ym, f)T, X′m=(x′m, y′m, f)T} (m=1, . . . , M) are known, H can be calculated. More specifically, for h=(H11, H12, H13, H21, H22, H23, H31, H32)T, the expression (2) is rewritten into a linear expression of h,
                                          (                                                                                xf                    ′                                                                                        yf                    ′                                                                                        ff                    ′                                                                    0                                                  0                                                  0                                                                      -                                          xx                      ′                                                                                                            -                                          yx                      ′                                                                                                                    0                                                  0                                                  0                                                                      xf                    ′                                                                                        yf                    ′                                                                                        ff                    ′                                                                                        -                                          xy                      ′                                                                                                            -                                          yy                      ′                                                                                            )                    ⁢          h                =                  (                                                                                          x                    ′                                    ⁢                  f                                                                                                                          y                    ′                                    ⁢                  f                                                              )                                    (        4        )            Substituting each set of corresponding points {Xm, X′m} into this expression, and solving the obtained simultaneous equations, h, that is, H can be calculated. From this, a method for calculating H not only uses the set of corresponding points but also detects the straight lines in the image and uses the line correspondences between the camera images is led as follows.
Now, suppose that together with the set of corresponding points {Xm, X′m}, a set of corresponding straight lines between camera images {anx+bny+cnf=0, a′nx′+b′ny′+c′nf′=0}(n=1, . . . , N) are obtained. Representing the straight line with the parameters In=(an, bn, cn)T, I′n=(a′n, b′n, c′n)T, each straight line on the image is represented such that,
                                          l            n            T                    ⁢          x                =                              0            ⁢                                                  ⁢                          l              n              ′T                        ⁢                          x              ′                                =          0                                    (        5        )            Since the expressions (5) and (2) hold at the same time, a projective transformation expression regarding the straight lineuIn=HTI′n  (6)is obtained (u is a non-zero constant). This expression (6) is expanded in the same manner as the expression (4). The simultaneous expressions of h obtained from the set of corresponding points {Xm, X′m} and the set of corresponding straight lines {In, I′n} are unified and finally represented such that,
                                          (                                                                                                      x                      1                                        ⁢                                          f                      ′                                                                                                                                  y                      1                                        ⁢                                          f                      ′                                                                                                            ff                    ′                                                                    0                                                  0                                                  0                                                                                            -                                              x                        1                                                              ⁢                                          x                      1                      ′                                                                                                                                  -                                              y                        1                                                              ⁢                                          x                      1                      ′                                                                                                                    0                                                  0                                                  0                                                                                            x                      1                                        ⁢                                          f                      1                      ′                                                                                                                                  y                      1                                        ⁢                                          f                      1                      ′                                                                                                            ff                    ′                                                                                                              -                                              x                        1                                                              ⁢                                          y                      1                      ′                                                                                                                                  -                                              y                        1                                                              ⁢                                          y                      1                      ′                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ⋮                                                                                                                                                                                                                                                                                                                                                                                                x                      M                                        ⁢                                          f                      ′                                                                                                                                  y                      M                                        ⁢                                          f                      ′                                                                                                            ff                    ′                                                                    0                                                  0                                                  0                                                                                            -                                              x                        M                                                              ⁢                                          x                      M                      ′                                                                                                                                  -                                              y                        M                                                              ⁢                                          x                      M                      ′                                                                                                                    0                                                  0                                                  0                                                                                            x                      M                                        ⁢                                          f                      ′                                                                                                                                  y                      M                                        ⁢                                          f                      ′                                                                                                            ff                    ′                                                                                                              -                                              x                        M                                                              ⁢                                          y                      M                      ′                                                                                                                                  -                                              y                        M                                                              ⁢                                          y                      ′                                                                                                                                                              a                      1                      ′                                        ⁢                                          c                      1                                                                                        0                                                                                            -                                              a                        1                        ′                                                              ⁢                                          a                      1                                                                                                                                  b                      1                      ′                                        ⁢                                          c                      1                                                                                        0                                                                                            -                                              b                        1                        ′                                                              ⁢                                          a                      1                                                                                                                                  c                      1                      ′                                        ⁢                                          c                      1                                                                                        0                                                                              0                                                                                            a                      1                      ′                                        ⁢                                          c                      1                                                                                                                                  -                                              a                        1                        ′                                                              ⁢                                          b                      1                                                                                        0                                                                                            b                      1                      ′                                        ⁢                                          c                      1                                                                                                                                  -                                              b                        1                        ′                                                              ⁢                                          b                      1                                                                                        0                                                                                            c                      1                      ′                                        ⁢                                          c                      1                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ⋮                                                                                                                                                                                                                                                                                                                                                                                                a                      N                      ′                                        ⁢                                          c                      N                                                                                        0                                                                                            -                                              a                        N                        ′                                                              ⁢                                          a                      N                                                                                                                                  b                      N                      ′                                        ⁢                                          c                      N                                                                                        0                                                                                            -                                              b                        N                        ′                                                              ⁢                                          a                      N                                                                                                                                  c                      N                      ′                                        ⁢                                          c                      N                                                                                        0                                                                              0                                                                                            a                      N                      ′                                        ⁢                                          c                      N                                                                                                                                  -                                              a                        N                        ′                                                              ⁢                                          b                      N                                                                                        0                                                                                            b                      N                      ′                                        ⁢                                          c                      N                                                                                                                                  -                                              b                        N                        ′                                                              ⁢                                          b                      N                                                                                        0                                                                                            c                      N                      ′                                        ⁢                                          c                      N                                                                                            )                    ⁢                                          ⁢          h                =                  (                                                                                          x                    1                    ′                                    ⁢                  f                                                                                                                          y                    1                    ′                                    ⁢                  f                                                                                    ⋮                                                                                                          x                    M                    ′                                    ⁢                  f                                                                                                                          y                    M                    ′                                    ⁢                  f                                                                                                                          a                    1                                    ⁢                                      c                    1                    ′                                                                                                                                            b                    1                                    ⁢                                      c                    1                    ′                                                                                                      ⋮                                                                                                          a                    N                                    ⁢                                      c                    N                    ′                                                                                                                                            b                    N                                    ⁢                                      c                    N                    ′                                                                                )                                    (        7        )            Solving this expression (7), h, that is, H is calculated. If M+N is greater than or equal to four, His calculated, employing various solving methods as described in “W. H. Press et al, Numerical Recipes in C, ISBN 0-521-43108-5”.
In order to obtain the projective matrix in the above manner, it is presumed that the set of corresponding points (pixels) and the set of straight lines between the camera images are obtained in advance.
However, many conventional image monitoring apparatuses required that a operator manually calculated the set of corresponding points and the set of corresponding straight lines when installing the image monitors, whereby the operator had to make the operation carefully without mistaking the correspondence, and calculate a great number of corresponding points and corresponding straight lines to acquire the transformation matrix at high precision, which took a lot of time.
When the image monitoring apparatus is mounted with some “deviation” in a place where vibration often occurs or uses a camera mounted on a vehicle, a situation may arise that is less favorable for a user who uses the image monitoring apparatus before recalibration is made by the coordinator, for example, the image monitoring apparatus may cause a malfunction, or the operator is needed to dispatch to the installed place.
As described above, to realize a high performance image monitoring apparatus or an obstacle detecting apparatus as described in JP-A-Hei.11-328365 and JP-A-2000-293693, it is required to make beforehand a stereo calibration for acquiring the image transformation parameters between the image pickup devices.
In the conventional stereo calibration apparatus, a manual operation of designating the corresponding points by the operator is essential, and enormous manpower cost and a lot of operation time are needed. Particularly, when a number of stereo calibrations are needed to be made, or the mounting of camera is unexpectedly changed, there is a problem that the manpower cost and the operation time are further increased.